Systems and Methods for Transmitting and Receiving Array Camera Image Data

ABSTRACT

Systems and methods for transmitting and receiving image data captured by an imager array including a plurality of focal planes are described. One embodiment of the invention includes capturing image data using a plurality of active focal planes in a camera module, where an image is formed on each active focal plane by a separate lens stack, generating lines of image data by interleaving the image data captured by the plurality of active focal planes, and transmitting the lines of image data and the additional data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application on U.S. patentapplication Ser. No. 13/668,057 filed Nov. 2, 2012, which is acontinuation application on U.S. patent application Ser. No. 13/470,252filed May 11, 2012, which application claims priority to U.S.Provisional Patent Application No. 61/484,920, filed May 11, 2011, thedisclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention is generally related to array cameras and morespecifically to the transmission of image data from an imager array toan external device such as a processor.

BACKGROUND OF THE INVENTION

In a typical camera, light enters through an opening (aperture) at oneend of the camera and is directed to a focal plane by a lens stack. Thelens stack creates an optical channel that forms an image of a sceneupon the focal plane. The focal plane includes an array of lightsensitive pixels, which are part of a sensor that generates signals uponreceiving light via the optical element. Commonly used sensors includeCCD (charge-coupled device) sensors and CMOS (complementarymetal-oxide-semiconductor) sensors.

Traditional cameras typically use a single focal plane to capture singleimages, one at a time. The image data from each pixel of the focal planeis then sent directly from the focal plane to a processor. The processorcan manipulate the image data, such as to encode the image data, storethe image data, forward the image data or modify the image data. In manyinstances, a standard interface is utilized between the focal plane andprocessor, which specifies the format in which data is transmittedbetween the focal plane and the processor.

The Mobile Industry Processor Interface Alliance (MIPI) is a non-profitcorporation that has promulgated interface specifications forconsistency to promote reuse and compatibility in mobile devices. MIPIhas created the Camera Serial Interface 2 (c) interface format (MIPIinterface format) for an interface between a camera and a processor.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiment of the invention areinvolve the transmission and/or reception of image data captured by acamera module including a plurality of focal planes. In severalembodiments, the image data is used to generate lines of image data thatinterleave image data captured by the pixels of different focal planes.In many embodiments, the sensor in the camera module also generatesadditional data describing the captured image data. In a number ofembodiments, the additional data is transmitted with the lines of imagedata and used by the interface circuitry on a device receiving the linesof image data to identify which pixels in the lines of image data areassociated with one or more of the plurality of images captured by thefocal planes of the camera module.

One embodiment of the invention includes capturing image data using aplurality of active focal planes in a camera module, where an image isformed on each active focal plane by a separate lens stack, generatinglines of image data by interleaving the image data captured by theplurality of active focal planes, and transmitting the lines of imagedata and the additional data.

A further embodiment also includes additional data providing informationfor deinterleaving a plurality of images from the captured image dataand transmitting the additional data with the lines of image data.

In another embodiment, the additional data describes the imager array.

In a still further embodiment, the additional data includes dataselected from the group of additional data describing: the number offocal planes in the imager array; the dimensions of the array of focalplanes in the imager array; the number of pixels in each dimension ofthe active focal planes; the integration time of the pixels in each ofthe active focal planes; and the gain of the pixels in each of theactive focal planes

In still another embodiment, the additional data describing the gain ofpixels includes additional data describing the analog gain of the pixelsin each of the active focal planes and additional data describing thedigital gain of the pixels in each of the active focal planes.

In a yet further embodiment, generating the lines of image data furthercomprises combining captured image data from the plurality of activefocal planes using a predetermined process.

In yet another embodiment, the predetermined process is selected from aplurality of predetermined processes for combining captured image data,and the additional data includes additional data indicating thepredetermined process used to combine the captured image data.

In a further embodiment again, each active focal plane comprises anarray of pixels including a plurality of rows of pixels that also form aplurality of columns of pixels, and the predetermined process forcombining captured image data comprises interleaving image data from arow selected from each of the active focal planes.

In another embodiment again, the predetermined process for combiningcaptured image data comprises using a modulo process, where the moduloprocess involves combining the image data captured by each of the pixelsin the selected rows of the active focal planes by interleavingpredetermined numbers of pixels from each of the selected rows.

In a further additional embodiment, the additional data describes atleast one of the lines of image data.

In another additional embodiment, the additional data includes dataselected from the group consisting of additional data describing: thetime at which each active focal plane started capturing the image data;and the row numbers of the rows in each of the active focal planes usedto capture the image data used to generate the line of image data.

A still yet further embodiment also includes packetizing the lines ofimage data and the additional data describing the captured image data toproduce at least one packet of data. In addition, transmitting the linesof image data and the additional data further comprises transmitting theat least one packet of data.

In still yet another embodiment the packet of data includes a packetheader containing at least a portion of the additional data and aplurality of lines of image data, and the additional data contained inthe packet header describes the imager array.

In a still further embodiment again, the packet of data further includesa packet footer indicating the end of the transmission of the pluralityof lines of image data.

In still another embodiment again, the packet of data comprises aplurality of lines of image data and a line header associated with eachof the plurality of lines of image data, and each line header includesadditional data describing the line of image data with which the lineheader is associated.

In a yet further embodiment again, the line head further comprisesadditional data describing the imager array.

Yet another embodiment again also includes pausing transmission duringframe blanking intervals, transmitting the plurality of lines of imagedata and the additional data describing the captured image data betweenframe intervals, and pausing during the transmission of the plurality oflines of image data and the additional data describing the capturedimage data during line blanking intervals.

A yet further additional embodiment also includes continuouslytransmitting the lines of image data and the additional data until allof the data is transmitted.

In yet another additional embodiment each lens stack in the cameramodule has a different field of view.

An embodiment of a method for receiving image data includes receivingimage data from a camera module including a plurality of focal planesusing interface circuitry, where the image data comprises lines of imagedata generated by interleaving pixels from a plurality of imagescaptured using the plurality of focal planes, and identifying the pixelsin the image data that are part of at least one of the plurality ofimages using the interface circuitry.

In a further embodiment of a method for receiving image data, the cameramodule also generates additional data describing the captured image dataand the method further includes receiving the additional data, andidentifying the pixels in the image data that are part of at least oneof the plurality of images using the interface circuitry furthercomprises identifying the pixels in the image data that are part of atleast one of the plurality of images using the additional data.

In another embodiment of a method for receiving image data, theadditional data includes data selected from the group consisting ofadditional data describing: the number of focal planes in the imagerarray; the dimensions of the array of focal planes in the imager array;the number of pixels in each dimension of the focal planes; theintegration time of the pixels in each of the focal planes; and the gainof the pixels in each of the focal planes.

In a yet further embodiment, the additional data describing the gain ofpixels includes additional data describing the analog gain of the pixelsin each of the active focal planes and additional data describing thedigital gain of the pixels in each of the active focal planes.

In yet another embodiment, the received lines of image data aregenerated using one of a plurality of predetermined processes, and theadditional data identifies the predetermined process used to generatethe lines of image data.

In a further embodiment again, the interface circuitry includes aprocessor configured to identify the pixels in the image data that arepart of at least one of the plurality of images.

An embodiment of a system for receiving image data in accordance with anembodiment of the invention includes interface circuitry configured toreceive image data from a camera module including a plurality of focalplanes, where the image data comprises lines of image data generated byinterleaving pixels from a plurality of images captured using theplurality of focal planes. In addition, the interface circuitry isfurther configured to identify the pixels in the image data that arepart of at least one of the plurality of images using the interfacecircuitry.

In a further embodiment of a system for receiving image data, the cameramodule also generates additional data describing the captured image dataand the interface circuitry is further configured to receive theadditional data, and identify the pixels in the image data that are partof at least one of the plurality of images using the additional data.

In another embodiment of a system for receiving image data, theadditional data includes data selected from the group of additional datadescribing: the number of focal planes in the imager array; thedimensions of the array of focal planes in the imager array; the numberof pixels in each dimension of the focal planes; the integration time ofthe pixels in each of the focal planes; and the gain of the pixels ineach of the focal planes.

In a still further embodiment, the additional data describing the gainof pixels includes additional data describing the analog gain of thepixels in each of the focal planes and additional data describing thedigital gain of the pixels in each of the focal planes.

In still another embodiment, the interface circuitry includes aprocessor configured to identify the pixels in the image data that arepart of at least one of the plurality of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an array camera in accordance with an embodiment ofthe invention.

FIG. 2 illustrates an imager array in accordance with an embodiment ofthe invention.

FIG. 3 illustrates image data transmitted as packets in accordance withan embodiment of the invention.

FIG. 4 illustrates transmission of image data as packets compatible withthe MIPI CSI-2 standard interface format in accordance with anembodiment of the invention.

FIG. 5 is an illustration of a transmission of a line of image data inaccordance with an embodiment of the invention.

FIG. 6 is a process for transmitting image data captured by an imagerarray in accordance with an embodiment of the invention.

FIGS. 7 a-7 c conceptually illustrate lines of image data transmitted byinterleaving image data from multiple focal planes using moduloprocesses in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for transmitting andreceiving image data captured by an imager array are illustrated. Inseveral embodiments, the imager array is configured to transmit imagedata to a processor and additional data describing the image data thatis being transmitted. A device receiving the image data can use theadditional data to reconstruct a plurality of images of a scene capturedby the imager array (often referred to as a light field). In severalembodiments, the image data and the additional data describing the imagedata are transmitted as a packet of image data that can include a packetheader as well as lines of image data, where each line of image data ispreceded by a line header. The lines of image data include the imagedata captured by focal planes of the imager array and can be utilized toconstruct a plurality of images of a scene. The line header can includedata identifying the particular pixels and focal planes that areincluded in a line of image data. In embodiments where a packet headeris present, the packet header can include embedded data that describesthe camera module to enable a processor to construct images from thepacket of image data.

In a variety of embodiments, a line of image data can correspond to arow, column or any organization of image data corresponding toparticular pixels from an image captured by one or more of the focalplanes in the imager array. In many embodiments, a line of image datacan include specific pixels from some or all of the focal planes of animager array.

All of the image data captured by a focal plane in the imager array canconstitute a low resolution image (the term low resolution here is usedonly to contrast with higher resolution images that can be synthesizedthrough super-resolution processing), which the processor can use incombination with other images captured by the imager array to constructa higher resolution image through super-resolution processing.Super-resolution processing is discussed in U.S. patent application Ser.No. 12/967,807 entitled “Systems and Methods for Synthesizing HighResolution Images Using Super-Resolution Processes”, filed Dec. 14,2010, the disclosure of which is hereby incorporated by reference in itsentirety. The processes for transmitting multiple images in accordancewith embodiments of the invention are more general, however, thantransmitting images to a processor for performing super-resolutionprocessing. Accordingly, processes in accordance with embodiments of theinvention can be used to transmit captured image data for multipleimages of a scene in a variety of applications including but not limitedto slow motion video applications involving registering of imagescaptured from different perspectives, and conventional videoapplications involving the capture of a sequence of sets of images tosynthesize a sequence of high resolution frames using super-resolutionprocesses.

In a number of embodiments, the imager array is configured to transmitimage data generated by focal planes via an interface format. Thecaptured image data is transmitted in accordance with the interfaceformat as a packet. These packets can be adapted to accord with anyinterface format, including but not limited to the MIPI CSI-2 interfaceformat (MIPI interface format).

Imager arrays, interface formats used to transfer captured image databetween imager arrays and processors, and processes for formattingcaptured image data into packets for transfer via an interface format inaccordance with embodiments of the invention are discussed furtherbelow.

System Architecture

Array cameras in accordance with many embodiments of the invention caninclude a camera module and a processor. The camera module can includean array of cameras. In various embodiments, an array camera includes acamera module and a processor. A camera module can include an imagerarray, which is a sensor that includes an array of focal planes. Eachfocal plane includes an array of pixels used to capture an image formedon the focal plane by a lens stack. The focal plane can be formed of,but is not limited to, traditional CIS (CMOS Image Sensor), CCD(charge-coupled device), high dynamic range sensor elements,multispectral sensor elements and various alternatives thereof. In manyembodiments, the pixels of each focal plane have similar physicalproperties and receive light through the same lens stack. Furthermore,the pixels in each focal plane may be associated with the same colorfilter. In a number of embodiments, at least one of the focal planesincludes a Bayer-pattern filter. In several embodiments, the focalplanes are independently controlled. In other embodiments, the operationof the focal planes in the imager array is controlled via a single setof controls.

An array camera in accordance with an embodiment of the invention isillustrated in FIG. 1. The array camera 100 includes a camera module 102that is configured to transmit (106) image data to a receiving device108 via an interface format involving the transmission of additionaldata describing the transmitted image data. The camera module 102includes an array of cameras 104. The cameras 104 in the camera module102 are formed from the combination of a lens stack and a focal plane.The camera module 102 can include an optic array of lens stacks and animager array of focal planes. These multiple cameras 104 may be activeor inactive at any given time. Array cameras are discussed in U.S.patent application Ser. No. 13/106,797 entitled “Architectures forimager arrays and array cameras” and U.S. patent application Ser. No.12/952,106 entitled “Capturing and processing of images using monolithiccamera array with heterogeneous imagers” the disclosure of bothapplications is hereby incorporated by reference in its entirety. Theimage data captured by these multiple cameras is sent from the focalplanes of each camera to a processor. The focal planes may havedifferent imaging characteristics, such as varying exposure times, starttimes, and end times. Therefore, the timing of the transmission of theimage data captured by each focal plane can vary. Accordingly, theimager array can transmit additional data describing the image data toenable a device receiving the image data to appropriately reconstructimages from the received image data.

In many embodiments, the array camera 100 captures images using aplurality of cameras 104, which can have different imagingcharacteristics. The array camera 100 can separately control each of thecameras to obtain enhanced image capture and/or to enhance processessuch as (but not limited to) super-resolution processes that may beapplied to the captured images. For example, each pixel of a focal planemay capture different wavelengths of light, or may capture the intensityof light, varying exposure times, start times, or end times. Once thearray camera 100 has commenced capturing image data using the pixels onthe imager array, the focal planes can commence transmitting the imagedata captured using the pixels to a receiving device 108. The image datacaptured by different camera modules can be interleaved for transmissionto a receiving device 108 that includes interface circuitry configuredto receive image data. In many embodiments, the interface circuitry isimplemented in hardware and/or using a processor. The receiving device108 can then organize the captured image data from the received packetand appropriately combine the image data to reconstruct the image(s)captured by one or more of the focal planes in the imager array.

In the illustrated embodiment, multiple images of a scene can becaptured by the camera module 102. As the image data is captured, thecamera module 102 transmits (106) the image data to a receiving device108. The camera module 102 transmits the image data using a small numberof local data storage cells on the camera module 102 that store thecaptured image data following capture by the cameras. The camera module102 manages the capture and transmission of image data so that thecaptured image data stored in the storage cells is transmitted by theimager array 102 in the time taken to capture and load the next set ofimage data into the storage cells. In this way, the camera module cancontinuously buffer and transmit image data using a number of local datastorage cells that is less than the total number of pixels in the cameramodule.

A line of image data transmitted by an imager array can be considered toequal the number of pixels in a row (column) of a focal plane multipliedby the number of focal planes. In several embodiments, the clockfrequency of transmitter circuitry on the imager array is set to adesired output data rate and the internal focal plane pixel rate is setto 1/N the desired output data rate (where N is the total number offocal planes). In many image transmission protocols, once a start ofline condition is sent, all of image data is transmitted withoutinterrupt until the end of line. Accordingly, a sufficient number ofdata storage cells and a buffering mechanism can be developed thatstarts transmission of pixels once there are sufficient pixels storedsuch that all of the pixels will have been captured and transmitted bythe time the end of the line of image data is reached. If, for example,an imager array including 16 focal planes (as in a 4×4 array) transmitsimage data from all focal planes, then there is very little data storageutilized prior to the start of focal plane readout, because the data istransmitted at approximately the rate that at which it is being read.If, however, the same imager array only has one active imager, thenalmost all of the pixels from a row (column) of the focal plane arestored since the buffer is being read 16 times as fast as it is beingwritten. Therefore, the data storage requirement would be one row ofpixels (i.e. 1/16^(th) of a line of image data). When eight focal planesare active, half the data from all eight focal planes is buffered beforetransmission commences to avoid underflow. Therefore, the total numberof data storage cells utilized is equal to four rows of pixels or onequarter of a line of image data. The above examples illustrate how thedata storage requirements of an imager array can vary based upon thenumber of active focal planes. In many embodiments, the total number ofstorage cells within an imager array is less than a quarter of a line ofimage data. In several embodiments, the total number of storage cellswithin an imager array is equal to a line of image data. In severalembodiments, the total number of data storage cells is between a quarterof a line of image data and a full line of image data. In a number ofembodiments, the total number of storage cells is equal to or greaterthan a line of image data. When the camera module transmits the capturedimage data, the incorporation of additional data describing the imagedata enables a peripheral device receiving the image data to reconstructthe images captured by each active camera in the imager array 102.

Imager arrays in accordance with many embodiments of the invention areconfigured to output image data via an interface format thataccommodates the transfer of image data captured via multiple focalplanes. In several embodiments, the imager array is configured totransmit captured image data in accordance with an interface format thatis compatible with standard interface formats, such as (but not limitedto) the MIPI CSI-2 interface format (MIPI interface format), the CameraLink interface format, and any of the Universal Serial Bus (USB)interface formats or FireWire interface formats. When image datacaptured from multiple focal planes is output by the imager array, thedevice receiving the image data is faced with the task of assembling theimage data into a plurality of images of a scene.

Although specific array camera system architectures are discussed above,any of a variety of system architectures for array cameras can beutilized as appropriate to the requirements of a specific application inaccordance with embodiments of the invention. The transmission of imagedata captured by a plurality of focal planes on an imager array andadditional data describing the image data that enables a receiver toreconstruct images from the image data in accordance with embodiments ofthe invention is discussed below.

Identification of Focal Planes

Due to the fact that imager arrays in accordance with many embodimentsof the invention can capture image data using more than one focal plane,processes for transmitting the captured image data include mechanismsfor identifying the specific focal plane used to capture a specific setof image data. A convention is typically adopted for identifyingspecific focal planes within an imager array. Focal planes on an imagerarray in accordance with an embodiment of the invention are conceptuallyillustrated in FIG. 2. The illustrated imager array 200 includes a 5×5array of focal planes 202. Each focal plane 202 includes an array ofpixels.

The focal planes on an imager array can be considered to be arranged ina matrix of “M” rows of focal planes, on a row axis notated as “m”, and“N” columns of focal planes, on a column axis notated as “n.” Individualfocal planes can be identified based upon their location within theimager array. In certain embodiments, a numbering convention is used inwhich the focal plane number is defined by the following equation:

Focal plane number=(m+(n*M))+1

where

-   -   m is the horizontal index into the imager array from 0 to M−1;    -   n is the vertical index into the imager array from 0 to N−1;    -   M is the total number of focal planes in the horizontal        direction; and    -   N is the total number of focal planes in the vertical direction.

In the illustrated embodiment, the horizontal index, or row, of focalplanes is numbered from 0 to 4. The vertical index, or column, of focalplanes is numbered from 0 to 4. Thereby, each row and column has 5 focalplanes for a total number of 25 focal planes in the illustrated array offocal planes. In many embodiments, the numbering starts at the top leftcorner of the imager array. In other embodiments, alternativeconventions are used to number the focal planes.

Although specific focal plane numbering conventions are discussed above,any of a variety of different conventions can be utilized to identify afocal plane. Given a specific convention, the identity of a focal planeutilized to capture specific image data can be transmitted with theimage data. Processes for transmitting image data with informationidentifying the focal plane that captured the image data are discussedfurther below.

Packetizing Image Data Captured by Multiple Focal Planes

In several embodiments, image data from a plurality of focal planes canbe packetized by inserting the image data and/or additional datadescribing the image data into a packet in such a way that a processorcan reconstruct images of a scene from the received image data. Aconceptual illustration of a packet including image data and additionaldata describing the image data transmitted by an imager array inaccordance with an embodiment of the invention is illustrated in FIG. 3.The packet 302 includes a packet header 304 and a packet footer 306. Thepacket 302 also includes a number of lines 308 of image data, where eachline 308 of image data includes a line header 310 and a line footer 312.In many embodiments, the packet header 304 and/or the line header 310contain additional data that describes the image data in such a way thata device receiving the packet can reconstruct a plurality of imagesusing image data including the lines of image data contained within thepacket. The number of lines of image data included within the packettypically depends upon the requirements of a specific application. Inmany embodiments, the packet can contain all of the lines of image datafor a single captured light field. The term light field can be used todescribe a number of two dimensional images that are captured of a scenefrom different perspectives. In other embodiments, the lines of imagedata of a captured light field can be divided and sent in multiplepackets. In many embodiments, image data captured by individual pixelsor groups of pixels from within a line of pixels from different focalplanes can be interleaved within a packet of image data.

In a number of embodiments, the packet header 304 contains embeddeddata. In many embodiments, the embedded data describes the camera modulefrom which image data is generated in such a way that a processor candetermine the structure of the image data in the packet and reconstructimages from the data received from the camera module. In severalembodiments, a packet header 304 includes embedded data such as (but notlimited to) the number of focal planes in an imager array, the timing ofthe image capture per focal plane, the identity of the particular focalplanes being read out, the total number of pixels in a focal plane, theresolution of an image taken by a focal plane, the timing of the pixelread outs and the gain for the focal plane. As discussed below, theembedded data described above need not be included in a packet headerand some or all of the information can be transmitted accompanying imagedata in different ways including but not limited to locating theadditional data elsewhere in the packet and/or transmitting theadditional data in a separate packet. Embedded data describing imagingdata in accordance with embodiments of the invention is discussedfurther below.

In the illustrated embodiment, the lines 308 of image data include lineheaders 310. The line header identifies the focal plane or focal planesand pixels in the imager array that captured the image data containedwithin the line of image data. A processor can utilize the line headerto identify the specific image data contained within the line 310 ofimage data. In various embodiments, a line header 310 includesinformation such as (but not limited to) the identify of the focal planethat captured the image data within the line and/or the identity of thespecific pixels(s) or group of pixels used to capture the image datacontained within the line of data, and a timestamp. Stated another way,a line of image data within a packet formatted in accordance withembodiments of the invention need not correspond to image data capturedusing a single line of pixels in a single focal plane. Indeed, packetsof image data in accordance with embodiments of the invention caninclude lines of image data containing image data captured by differentlines of pixels and/or from different focal planes. Inclusion of theadditional data describing the line of image data in the line headerallows a processor to receive and process image data from multipleimages multiplexed into a single packet or stream of packets. Differenttypes of embedded data that can be included in line headers (orelsewhere) in accordance with embodiments of the invention are discussedfurther below.

Each line 308 of image data can include a line footer 312 to indicatethat the line of image data 308 associated with the preceding lineheader 310 has ended. Also, each packet 302 can include a packet footer306 to indicate that the image data associated with the previous packetheader 304 has ended. In many embodiments, the imager array isconfigured to generate multiple packets 302 to contain the image datacaptured by the focal planes and each packet includes multiple lines ofimage data.

Due to the manner in which image data is captured by different sets ofpixels in different focal planes as data is transmitted by the imagerarray, the processor typically cannot predict the order in which it willreceive image data from the imager array. In many embodiments, theprocessor has no knowledge of the focal plane or focal planes thatcaptured the image data contained within a line of image data withoutreference to the packet header and/or the line header for the line ofimage data. However, in other embodiments the imager array imposesconstraints with respect to the order in which image data is captured byspecific focal planes (see for example the discussion below with respectto FIGS. 7 a-7 c) and a processor can rely upon the predetermined orderof image data capture to reconstruct the image data. While imposingconstraints on the order in which image data is captured can reduce theflexibility of the image array with respect to the manner in which imagedata is captured from different focal planes, the predictable manner inwhich image data is received from the imager array can result in thereduction in the amount of additional data transmitted in conjunctionwith the image data by removing information that identifies the focalplane and/or pixels that captured the image data. In many embodiments,the manner in which the imager array is constrained to capture imagedata enables the packet header, the packet footer, the line headerand/or the line footer illustrated in FIG. 3 to be eliminated.

Although the inclusion of specific pieces of information within packetheaders and/or line headers is described above, any information thatenables the reconstruction of multiple images from image datamultiplexed into a single packet or stream of packets can beincorporated into a packet of image data in accordance with embodimentsof the invention. Transmission of image data compatible with the MIPIinterface format is discussed further below.

Image Data Transmission Compatible with the MIPI Interface Format

In several embodiments, imager arrays transmit image data and additionaldata describing the image data in a manner that is compatible with anexisting interface format for the transmission of image data by aconventional camera including a single focal plane. A conceptualillustration of image data and additional data describing the image datatransmitted as packets compatible with the MIPI CSI-2 standard interfaceformat (MIPI interface format) in accordance with an embodiment of theinvention is illustrated in FIG. 4. The conceptual illustration can beread as involving transmissions from left to right in the X direction420 and from top to bottom in the Y direction 422. The transmissionbegins with a MIPI frame blanking interval 402. A MIPI frame start (MFS)410 indicator is then sent by the imager array, followed by a portion ofthe MIPI header 412. A packet of data generated in accordance withembodiments of the invention is inserted within the standard MIPIcontainer as embedded data. Accordingly, the first line of data withinthe MIPI container can include a packet header 424 containinginformation concerning the focal planes that generated the image data(see discussion above).

The transmission of the first line of the MIPI container is completed bythe transmission of a MIPI footer 406. There is a pause during the MIPIline blanking interval 408, and then the next portion of the MIPI header412 is transmitted. The next line of the MIPI container includes a lineof image data 414. In embodiments where the order in which the lines ofimage data transmitted by the imager array is not predetermined, theline of image data can be preceded by a line header and followed by aline footer. In embodiments where the lines of image data aretransmitted in a predetermined order (see for example the discussion ofFIGS. 7 a-7 c), a line header and/or line footer may not be utilized.

The process of transmitting a MIPI footer, pausing during a MIPI lineblanking interval, transmitting a portion of the MIPI header, andtransmitting lines of image data within the MIPI container continuesuntil all the lines of image data in the packet are transmitted. Inseveral embodiments, an embedded packet footer is transmitted in theMIPI container to indicate that the transmission of the packet iscomplete. Following the transmission of the packet, the transmission ofthe MIPI container is completed by transmitting a MIPI footer 406 and aMIPI frame end 416. Although the packet illustrated in FIG. 4 involvestransmitting one line of image data between line blanking intervals ofthe MIPI container, in many embodiments the packet header and the linesof image data do not correspond with the line blanking intervals of theMIPI container. Stated another way, a single line blanking interval ofthe MIPI container can contain image data from two or more lines ofimage data. Accordingly, the line headers and/or line footers areutilized to identify the individual lines of image data within thecontainer.

As can readily be appreciated, the process illustrated in FIG. 4involves formatting a packet of data including image data and additionaldata describing the image data generated in accordance with embodimentsof the invention within a conventional MIPI container. In this way, animager array can utilize an interface standard developed to enable thetransmission of image data captured by a single focal plane to enabletransmission of a packet of data containing image data captured by aplurality of focal planes (i.e. a light field). In other embodiments,similar processes can be utilized to transmit packets formatted in themanner outlined above using other containers and/or interface formatsincluding (but not limited to) a CameraLink interface format, a USBinterface format, or a Firewire interface format.

Additional Data Describing an Imager Array

Imager arrays in accordance with many embodiments of the inventionadditional data describing the imager array that can be embedded in apacket header or elsewhere in image data transmitted by the imagerarray. A processor can utilize the additional data describing the imagerarray to reconstruct images from the image data contained within thepacket. In various embodiments, the additional data describing theimager array is divided into fixed descriptors and dynamic descriptors.Fixed descriptors are pieces of data within a packet header that arefixed values that are specific to an array camera configuration, andwhich do not change during array camera operation. Dynamic descriptorsare pieces of data within the embedded data that can vary during arraycamera operation. In many embodiments, dynamic descriptors areparameters describing the operational states such as (but not limitedto) focal plane exposure time or focal plane processing gain. In severalembodiments, an imager array transmits image data and additional dataincluding at least one of the total number of focal planes, the totalnumber of pixels being read out per focal plane in a first dimension,the total number of pixels being read out per focal plane in a seconddimension, the integration time of the pixels in each focal plane, andthe gain applied to each pixel in each focal plain (which can be dividedinto analog gain and digital gain). The specific data transmitteddescribing the imager array typically depends upon the information thatis provided to the receiving device about the imager array by othersources. In many embodiments, imaging characteristics such as theintegration time can be specified by a receiving device such as aprocessor and the information is transmitted by the imager array tocommunicate to the receiving device that the imaging characteristics ofthe relevant focal planes have been updated in response to theinstruction. In many embodiments, an imager array transmits image dataand the gain applied to each pixel in each focal plain (which can bedivided into analog gain and digital gain), and additional dataincluding the integration time of the pixels in the active focal planesand at least one of the total number of focal planes, the total numberof pixels being read out per focal plane in a first dimension, the totalnumber of pixels being read out per focal plane in a second dimension.In other embodiments, any of a variety of combinations of informationincluding additional pieces of information can be transmitted asadditional data describing image data in accordance with embodiments ofthe invention.

In many embodiments, a transmitter within an imager array includesadditional data describing the imager array in the packet header toallow the receiver within a processor to determine the structure of thedata in a packet in order for the software in the processor toreconstruct the image data from the camera. The additional datadescribing the imager array in the packet header is largely dependentupon the specific application and array camera configuration.

In several embodiments, the additional data describing the imager arrayis transmitted as embedded data within a MIPI container using theembedded 8-bit non image data (data-type=0x12) format specified as partof the MIPI interface standard. The following table provides thestructure of various fixed descriptors that can be utilized inadditional data describing an imager array transmitted in accordancewith many embodiments of the invention. Unless otherwise stated the byteorder is MSB first. In several embodiments, one, multiple, and/or all ofthe following fixed descriptors can be included as embedded data withina packet header:

Length Name (bytes) Description Data ID 5 Fixed field start of embeddeddata. Value is ASCII character string “PICAM”. Device ID 4 Deviceidentification word. A product specific code. Revision 2 Siliconrevision word. Describes the ID version of the silicon Manufac- 8Manufacturer identifier. turer ID Array 1 Number of focal planes in Mand N, format where M is the number of focal planes per M, N row in theimager array and N is the number of focal planes per column in theimager array. Upper 4 bits denote number of the focal planes along the Xaxis. Start of focal 2 The frequency used to generate the start planeimage of focal plane image data capture time tag data capture counterfrequency values. The frequency time tag is represented using a 16-bitfixed counter precision 8.8 format (0 to 255.99609 MHz frequency in0.00390625 MHz steps).

The following table provides the structure of various dynamicdescriptors that can be utilized in the additional data describing animager array transmitted in accordance with embodiments of theinvention. Unless otherwise stated the byte order is MSB first. Inseveral embodiments, one, multiple, and/or all of the following dynamicdescriptors can be included as embedded data within a packet header:

Length Name (bytes) Description Active Focal 8 One hot encoding of whichfocal planes Planes are being readout in the upcoming packet. Thisprovides information on which focal planes are active and providingimage data. Supporting up to 64 focal planes. Each bit corresponds to asingle focal plane within the array with a 1 indicating a given focalplane is being readout. The bit assignment corresponds to the scalarversion of the focal plane number format. Total 2*M*N Number of pixelsbeing readout per focal pixels X plane in X. Each focal plane has a 16bit word. The focal plane ordering corresponds to the scalar version ofthe focal plane number format. Inactive focal plane (focal plane notpart of the readout) are set to 0x0000h. Total 2*M*N Number of pixelsbeing readout per focal pixels Y plane in Y. The focal plane orderingcorresponds to the scalar version of the focal plane numbering format.Inactive focal planes (focal planes not part of the readout) are set to0x0000h. Integration 3*M*N Integration time per focal plane Time(corresponding to the first row of image data in the upcoming packet) inunits of microseconds. The focal plane ordering corresponds to thescalar version of the focal plane numbering format. Inactive focalplanes (focal planes not part of the readout) are set to 0x000000h.Analog 1*M*N Analog Gain per focal plane Gain (corresponding to thefirst row of image data in the upcoming focal plane) in linear units.The gain is represented using an 8 bit fixed point 5.3 format (0 to31.875x in 0.125 steps). The focal plane ordering corresponds to thescalar version of focal plane numbering format. Inactive focal planes(focal planes not part of the readout) are set to 0x00h. Digital 1*M*NDigital gain per focal plane Gain (corresponding to the first row ofimage data in the upcoming focal plane) in linear units. The gain isrepresented using a 7 bit fixed point 3.4 format. The focal planeordering corresponds to the scalar version of focal plane numberingformat. Inactive focal planes (focal planes not part of the readout) areset to 0x00h. Packing 2 16-bit integer defining the packing Modulomodulo of the pixel data (see discussion of FIGS. 7a-7c). Modulo 0 canmean modulo equal Total Pixels X (see above). User Variable Reserved forsystem capability expansion. Padding Variable Expands the length of theembedded data section to be equal to a line of the image data with apreceding line header and trailing line footer. The value of paddingbytes is not specified.

Referring to the above tables, each of the descriptors is discussedfurther below. Furthermore, although a specific number of bytes are usedin embodiments throughout the specification for fixed and dynamicdescriptors, any number of bytes may be utilized in other embodiments asappropriate to specific applications.

“Data identification (ID)” is a fixed descriptor of 5 bytes that can beused at the start of the embedded data. The value of which can berepresented in an ASCII character string such as (but not limited to)“PICAM”.

“Device ID” is a fixed descriptor of 4 bytes that can be used as adevice identification word.

“Revision ID” is a fixed descriptor of two bytes in length. “RevisionID” is a silicon revision word and describes the version of the imagerarray.

“Manufacturer ID” is a fixed descriptor of 8 bytes in length.“Manufacturer ID” is a manufacturer identifier.

“Array format, M, N” is a fixed descriptor of 1 byte in length. “Arrayformat” describes the number of focal planes in the imager array. Inseveral embodiments, the array format specifies values M and N, where Mis the number of focal planes per row in the imager array and N is thenumber of focal plane per column in the imager array.

“Start of focal plane image data capture time tag counter frequency” isa fixed descriptor of 2 bytes in length that indicates the frequencyused to generate the start of focal plane image data capture time tagcounter frequency values. In several embodiments, the frequency isrepresented using a 16-bit fixed precision 8.8 format (0 to 255.99609MHz in 0.00390625 MHz steps). In other embodiments, any of a variety ofdata values can be utilized to represent the packet time tag values withprecision appropriate to the specific application.

“Active focal plane” is a dynamic descriptor of 8 bytes in length. The“active focal plane” descriptor describes the focal planes that are readout in the upcoming packet. Certain embodiments provide for support ofas many as 64 focal planes. In other embodiments, the number of focalplanes can exceed 64 focal planes and the embedded data indicates theappropriate number of focal planes. Each bit corresponds to a singlefocal plane within the imager array with a 1 indicating that the packetcontains image data captured using the focal plane.

“Total pixels X” is a dynamic descriptor of 2*M*N in length thatdescribes the number of pixels being read out per focal plane in the Xaxis direction. In several embodiments, each pixel has a 16 bit wordassociated with it. Inactive focal planes (focal planes not part of thereadout) can be set to 0x0000h.

“Total pixels Y” is a dynamic descriptor of 2*M*N in length thatdescribes the number of pixels being readout per focal plane along the Yaxis. Inactive focal planes (focal planes not part of the readout) canbe set to 0x0000h.

“Integration Time” is a dynamic descriptor of 3*M*N in length thatdescribes the integration time per focal plane (corresponding to thefirst row of data in the upcoming packet) in units of microseconds.Inactive focal planes (focal planes not part of the readout) are set to0x000000h.

“Analog Gain” is a dynamic descriptor of 1*M*N in length that describesthe analog gain per focal plane (corresponding to the first row of datain the upcoming packet) in linear units. In several embodiments, thegain is represented using an 8 bit fixed point 5.3 format (0 to 31.875xin 0.125 steps) value. Inactive focal planes (focal planes not part ofthe readout) are set to 0x00h.

“Digital Gain” is a dynamic descriptor of 1*M*N in length that describesthe digital gain per focal plane (corresponding to the first row of datain the upcoming packet) in linear units. In several embodiments, thegain is represented using an 7 bit fixed point 3.4 format. Inactivefocal planes (focal planes not part of the readout) are set to 0x00h.

“Packing Modulo” is a dynamic descriptor of 2 bytes length that definesthe packing modulo process used to interleave pixel data from the activefocal planes into a line of image data.

“User” is a dynamic descriptor of variable length that can be utilizedto support system capability expansion.

“Padding” is a dynamic descriptor of variable length that allows forexpansion of the length of the embedded data section to be equal to aline of the image data with preceding line header and trailing linefooter. The value of padding bytes is not specified and is variabledepending on the needs of the particular data section.

Although specific pieces of additional data describing an imager arrayin specific formats are disclosed above, the additional data describingan imager array (if any) transmitted by an imager array in accordancewith an embodiment of the invention depends upon the requirements of aspecific application. In many embodiments, a combination of the abovepieces of additional data and/or other pieces of additional datadescribing the imager array are transmitted so that a processorreceiving the information can reconstruct images from the image datacaptured by a imager array in accordance with embodiments of theinvention. Accordingly, the manner in which image data is transmittedbetween the imager array and the processors is not limited in any way totransmissions that include a specific set of additional data describingthe imager array. Specific sets of additional data describing an imagerarray including (but not limited to) one or more of the dynamicdescriptors identified above can, however, facilitate the reconstructionof images by a processor.

Additional Data Describing a Line of Image Data

Imager arrays in accordance with many embodiments of the inventionformat image data captured by groups of pixels within one or more focalplanes into a line of image data and can transmit the line of image datawith additional data describing the line of image data to other devices.A single line of image data typically corresponds to image data capturedby a row or a column of pixels within each active focal plane in animager array. In many embodiments, however, a line of image data cancorrespond to image data captured from pixels in one or more rows of oneor more active focal planes (see discussion of FIGS. 7 a-7 c below). Inthe context of the transmission of packets of data illustrated in FIG.4, lines of image data (possibly including a line header containingadditional data describing the line of image data and a line footer) aretransmitted within a MIPI container between line blanking intervals.

The transmission of a line of image data and a line header containingadditional data describing the line of image data by an imager array inaccordance with an embodiment of the invention is conceptuallyillustrated in FIG. 5. The line of image data 500 begins with a lineheader 502 containing additional data that describes the line of imagedata, followed by captured image data 504, and terminates with a footerindicating that the particular line of image data is complete. Referringback to FIG. 4, when a line of image data is transmitted in accordancewith a transmission format similar to the MIPI interface format, theline header and line footer are located within the data 414 (i.e. aredistinct from the packet header (PH) 424 and the packet footer (PF)426). Although much of the discussion refers to the inclusion ofadditional data describing a line of image data in the line header, inmany embodiments additional data describing the imager array can also beincluded in a line header. Furthermore, additional informationdescribing a line of image data need not be located within a line headerand can be communicated elsewhere with respect to the image data.

A variety of different control schemes can be utilized to control thetiming of the capture of image data by different pixels within the focalplanes of an imager array and with respect to the transmission of thecaptured image data by the imager array. In several embodiments, imagedata from a specific focal plane is captured one row (or column) ofpixels at a time. As noted above, the focal planes in an imager arraycan be separately controlled. Therefore, the sequence with which rows ofimage data are read from each focal plane can vary depending uponcontrol parameters including, but not limited to, exposure timeassociated with each focal plane. When image data is captured by thepixels, the image data is read out. The image data is not typicallytransmitted directly from the pixels but is buffered. The amount of datathat is buffered typically depends upon the rate at which image data iscaptured and the rate with which image data can be transmitted off-chipby the imager array. As discussed above, the amount of buffered pixeldata can also depend upon the number of active focal planes.

In numerous embodiments, a line of image data may be a row of pixel datafrom each active focal plane, a column of pixel data from each activefocal plane or any other grouping of image data captured by pixelswithin one or more focal planes. Accordingly, the term “line” is ageneric term. In many embodiments, the line of image data includes imagedata captured from pixels in a single focal plane. In other embodiments,the line of image data includes image data captured from multiple focalplanes. Additional data describing the line of image data allowssoftware or hardware on a receiver to determine from which focalplane(s), which row/column and/or which pixels on that focal planecaptured specific image data within a line of image data.

In several embodiments, the image data is transmitted using the RAW8 orRAW10 data formats defined within the MIPI interface format, althoughother data formats can be supported as necessary including formatsinvolving 8 bits per pixel and formats involving 10 bits per pixel.

In a variety of embodiments, the size of image data with accompanyingline headers containing additional data describing the image datatransmitted by a imager array when capture of a light field occurs (i.e.including all the image data associated with the images of the scenecaptured by each of the focal planes in the imager array) in bytes canbe calculated using the following equation:

((X*N)+P)*Y*k

where

-   -   X is the total pixels in an row of pixels within a specific        focal plane,    -   Y is the total pixels along a Y axis,    -   N is the number of focal planes in an array,    -   P is the line header length, and    -   k is a scale factor that can be used to account for 10 bits per        pixel transmission (where 8 bits per pixel (bpp) yields k=1 and        10 bpp yields k=1.25).

In the case of 10 bits per pixel transmission, each line of data ispadded with additional pixels to provide an integer multiple of 4.Padding pixels can have a value of 0x3Fh.

In many embodiments, the additional data describing a line of image datacan include one or more of timing information concerning the frame towhich the image data from each focal plane belongs, and the row (column)number in each focal plane from which image data in the line of imagedata is read out. In a number of embodiments, each line of data ispreceded by a line header containing the additional informationdescribing the line of image data, which provides the processor with thenecessary information about the line data to reconstruct the imagescaptured by the focal planes. The following table provides additionaldata that can be included in a line header in accordance with anembodiment of the invention to enable reconstruction of images by aprocessor:

Length Name (Bytes) Description Start of <=4*M*N A (nominally) 32-bitnumber for each focal focal plane corresponding the relative start timeplane im- for the readout of the focal plane. The age data start time ofa focal plane capturing image capture data is captured from a freerunning 32 bit time tag counter internal to the transmitter counting inunits of clock cycles where the frequency of the counter is defined inthe embedded image data line header. The focal plane orderingcorresponds to the scalar version of the focal plane numbering format.Inactive focal planes (as defined in the active focal planes field ofthe embedded data in the packet header) shall have their start of focalplane image data capture time tag omitted. Note in 10 bits per pixeltransmission mode the individual bytes are expanded to 10 bits andpadded with 2 LSBs = 00. Row <=2*M*N A (nominally) 16-bit line numberfor each Numbers focal plane indicating from which physical row in thefocal plane the subsequent data is from. Nonphysical rows (virtual rows)that may be transmitted can have their row number set to 0xffffh. Thefocal plane ordering corresponds to the scalar version of the focalplane numbering format. Inactive focal planes (as defined in the activefocal planes field in the embedded data of the packet header) shall havetheir line number omitted. Note in 10 bits per pixel transmission modethe individual bytes are expanded to 10 bits and padded with 2 LSBs =00. Pad- Variable Variable padding range to ensure the line ding (0 or2) header falls on a modulo 5 byte boundary in the case of 10 bits perpixel transmission. Padding words shall have the value of 0x3Fh

Referring to the above table, each of the descriptors is discussedfurther below. Furthermore, although a specific number of bytes are usedin embodiments throughout the specification for fixed and dynamicdescriptors, any number of bytes may be utilized in other embodiments asappropriate to specific applications.

“Start of focal plane image data capture time tag” is equal to or lessthan 4*M*N bytes in length and describes a (nominally) 32 bit number foreach focal plane corresponding the relative start time for the readoutof the focal plane. The start of the image data associated with a pixelcan be captured from a free running 32 bit counter internal to thetransmitter within the imager array counting in units of clock cycleswhere the frequency of the counter is defined in the embedded data.Inactive focal planes (as defined in the active focal plane field of theembedded data of the packet header) can have their start of focal planeimage data capture time tag omitted. Note in 10 bits per pixeltransmission mode the individual bytes are expanded to 10 bits andpadded with 2 LSBs=00.

“Row Numbers” is equal to or less than 2*M*N bytes in length anddescribes a (nominally) 16 bit line number for each focal planeindicating from which physical row or line in the focal plane thesubsequent image data is from (the appropriate number of bits dependsupon the number of lines of pixels and need not be 16 bits). Nonphysicalrows or lines (virtual rows or lines) that may be transmitted can havetheir row or line number set to 0xffffh. Inactive focal planes (asdefined in the active focal plane field of the embedded data) can havetheir line number omitted. In 10 bits per pixel transmission mode, theindividual bytes are expanded to 10 bits and padded with 2 LSBs=00.

“Padding” is of variable bytes in length (such as 0 or 2 in certainembodiments). A variable padding range enables the line header to fallon a modulo 5 byte boundary in the case of 10 bits per pixeltransmission. In several embodiments, padding words have the value of0x3Fh.

Although specific pieces of additional data describing a line of imagedata in specific formats are disclosed above, the additional datadescribing a line of image data (if any) transmitted accompanying a lineof image data by a imager array in accordance with an embodiment of theinvention depends upon the requirements of a specific application. Inmany embodiments, a combination of the above pieces of additional dataand/or other pieces of additional data are transmitted in a line headerso that a processor receiving the additional data describing the linesof image data can reconstruct images from the image data captured by aimager array in accordance with embodiments of the invention.Accordingly, the manner in which data is transmitted between the imagerarray and the processor is not limited in any way to transmissions thatinclude a specific set of additional data describing lines of imagedata. Specific sets of additional data describing lines of image datacan, however, facilitate the reconstruction of images by a processor.

Including Additional Data Describing Imager Array in Line Headers

As discussed above, additional data describing an imager array may beutilized to allow a receiver to determine the structure of image data ina packet and reconstruct images. In certain embodiments, the additionaldata describing an imager array can be included in a line header thatmay also contain additional data describing a line of image data. Thiscan be advantageous in certain embodiments where the packet headerlength may be limited or the separate transmission of additionalinformation describing an imager array is not desirable.

The following table includes additional data describing an imager arraythat can be included in a line header in accordance with an embodimentof the invention to enable reconstruction of images by a processor:

Length Name (Bytes) Description Free 2 Fixed value of ASCII code for“FR”. Note in running 10 bits per pixel transmission mode the modeindividual bytes are expanded to 10 bits and padded with 2 LSBs = 00.Packet 2 Arbitrary packet number counter. Initialized counter to zeroupon reset. Incremented by one for every packet transmitted inaccordance with the MIPI interface format. Wraps around at 0xffff. Notein 10 bits per pixel transmission mode the individual bytes are expandedto 10 bits and padded with 2 LSBs = 00. Row <=2*M*N A (nominally) 16-bitline number for each number focal plane indicating from which physicalrow in the focal plane the subsequent data is from. Nonphysical rows(virtual rows) that may be transmitted can have their row number set to0xffffh. The focal plane ordering corresponds to the scalar version ofthe focal plane numbering format. Inactive focal planes (as defined inthe active focal planes field of the embedded data) shall have theirline number omitted. Note in 10 bits per pixel transmission mode theindividual bytes are expanded to 10 bits and padded with 2 LSBs = 00.Pad- Variable Variable padding range to ensure the line ding (0 or 2)header falls on a modulo 5 byte boundary in the case of 10 bits perpixel transmission. Padding words shall have the value of 0x3Fh

Referring to the above table, each of the descriptors is discussedfurther below. Furthermore, although a specific number of bytes are usedin embodiments throughout the specification for fixed and dynamicdescriptors, any number of bytes may be utilized in other embodiments asappropriate to specific applications.

“Free running mode” is equal to 2 bytes in length and describes a fixedvalue of ASCII code for “FR”. Note in 10 bits per pixel transmissionmode the individual bytes are expanded to 10 bits and padded with 2LSBs=00.

“Packet counter” is equal to 2 bytes in length and describes anarbitrary packet number counter. This may be initialized to zero uponreset, incremented by one for every packet transmitted in the MIPIinterface format and wraps around at 0xffff. Note in 10 bits per pixeltransmission mode the individual bytes are expanded to 10 bits andpadded with 2 LSBs=00.

“Row Number” is equal to or less than 2*M*N bytes in length anddescribes a (nominally) 16 bit row (column) number for each focal planeindicating from which physical row or line in the focal plane thesubsequent image data is from (the appropriate number of bits dependsupon the number of lines of pixels and need not be 16 bits). Nonphysicalrows or lines (virtual rows or lines) that may be transmitted can havetheir row or line number set to 0xffffh. Inactive focal planes (asdefined in the active focal plane field of the embedded data) can havetheir line number omitted. In 10 bits per pixel transmission mode, theindividual bytes are expanded to 10 bits and padded with 2 LSBs=00.

“Padding” is of variable bytes in length (such as 0 or 2 in certainembodiments). A variable padding range enables the line header to fallon a modulo 5 byte boundary in the case of 10 bits per pixeltransmission. In several embodiments, padding words have the value of0x3Fh.

Although specific pieces of additional data describing an imager arraythat can be included in a line header are disclosed above, theadditional data transmitted with a line of image by an imager array inaccordance with an embodiment of the invention depends upon therequirements of a specific application. In many embodiments, acombination of the above pieces of additional data describing the imagerarray and/or other pieces of additional data describing the imager arrayand/or one or more lines of image data are transmitted in a line headerso that a processor receiving the information can reconstruct imagesfrom the image data captured by the imager array. Accordingly, themanner in which data is transmitted between the imager array and areceiver is not limited in any way to transmissions that include aspecific set of additional data in the line headers or packet headers.

Light Field Capture and Transmission Processes

Imager arrays in accordance with many embodiments of the invention cancapture image data using multiple focal planes and transmit the imagedata and additional information describing the image data to aprocessor. As noted above, the need to transmit the image data andadditional data describing the image data within a specific containermay influence the manner in which the data is transmitted. As notedabove, the additional data describing the image data typically dependsupon the manner in which the image data is captured and thecharacteristics of the imager array used to capture the light field.

A process for capturing a light field and transmitting the image dataand additional data describing the image data to an external device suchas (but not limited to) a processor using an imager array in accordancewith an embodiment of the invention is illustrated in FIG. 6. Theprocess 600 can commence by transmitting a packet header (602) thatincludes additional data describing the imager array. As noted above,the transmission of a packet header can be optional. During the process,image data is captured (604) by the pixels in the imager array. Invarious embodiments, capturing (604) image data involves controlling thepixels and pixel settings used when capturing the image data. In certainembodiments, the pixel settings can be performed globally per focalplane. In many embodiments, the process also includes resetting thepixels after the capture of image data.

In many embodiments, a line header including additional data describingthe a line of image data is transmitted (606) by the imager array, andthen a line of captured image data is transmitted (608). As noted above,the line of image data can include image data from one or more focalplanes and the additional data can identify the groups of pixels and/orthe focal planes that captured the image data within the line of imagedata. Upon completion of the transmission of a line of image data, aline footer can optionally be transmitted (610) by the imager array toindicate that the transmission of the line of image data is complete.The process repeats until a determination (612) is made that the captureof the light field (i.e. a plurality of two dimensional images capturedfrom different viewpoints) using the plurality of focal planes iscomplete. Following the completion of the transmission of image data bythe imager array, the imager array can optionally transmit (614) apacket footer that indicates the completion of the packet of data. Asnoted above, the specific manner in which the image data and theadditional data describing the image data is transmitted is oftendictated by an interface format (e.g. the MIPI interface format).

In several embodiments, line headers, line footers and/or packet footersare omitted based on the image data being configured to capture apredetermined amount of image data and/or the transmission of thecaptured image data in a predetermined sequence. Although a specificprocess for transmitting image data and additional data describing theimage data is discussed above with respect to the process illustrated inFIG. 6, any of a variety of processes that enable the transmission oflines of image data captured using different focal planes and additionaldata that can be utilized to construct a plurality of images from thelines of image data can be utilized in accordance with embodiments ofthe invention. Processes for transmitting image data captured in apredetermined manner by an imager array are discussed further below.

Interleaving Image Data from Multiple Focal Planes

In a number of embodiments, image data from multiple focal planes can beinterleaved in a predetermined manner to simplify the transmission ofimage data and reduce the amount of additional data describing the imagedata that is transmitted accompanying the image data. In manyembodiments, imager arrays use a “modulo” scheme for interleaving thecaptured image data into a packet in a predetermined manner. A moduloscheme involves capturing and transmitting image data from the samegroup of pixels in each active focal plane (or each focal plane in agroup of focal planes) within an imager array, such as the same row,rows, column, or columns. In other embodiments, a modulo scheme caninvolve specifying the specific row (or column) from a set of focalplanes and then reading out corresponding pixels or groups of pixelsfrom each specified row (or column).

Three different modulo processes for capturing and transmitting imagedata in accordance with embodiments of the invention are illustrated inFIGS. 7 a-7 c. A line of image data transmitted in accordance with afirst modulo process 726 is illustrated in FIG. 7 a and involvestransmitting the first pixel 702 from a row selected from each of atotal of twenty focal planes before transmitting the second pixel 704from the selected row from each of the 20 focal planes. Accordingly, theprocess illustrated in FIG. 7 a can be referred to as a modulo oneprocess. As can readily be appreciated, the number of focal planes isarbitrary. In several embodiments, the specific rows transmitted by eachfocal plane can be specified in additional data contained within a lineheader transmitted prior to the transmission of the image data.

Similarly, a line of image data transmitted in accordance with a secondmodulo process 728 is conceptually illustrated in FIG. 7 b and involvestransmitting pairs of pixels from a selected row in each of the activefocal planes. In the illustrated embodiment, the first 706 and second708 pixels from a selected row in each of a total of twenty focal planesis transmitted before transmitting the third 710 and fourth 712 pixelsof the selected rows. The process repeats transmitting pairs of pixelsfrom the selected row in each focal plane until the transmission of theimage data captured by the selected rows of pixels is complete.Accordingly, the process illustrated in FIG. 7 b can be referred to as amodulo two process.

A line of image data transmitted in accordance with a third moduloprocess 730 is conceptually illustrated in FIG. 7 c and involvestransmitting the first 714, second 716 and third 718 pixels of a rowselected from each of a total of twenty active focal planes beforetransmitting the fourth 720, fifth 722 and sixth pixels (not shown) fromeach of the selected rows. The process repeats transmitting sets ofthree pixels from the relevant row in each focal plane until thetransmission of the image data captured by the selected rows of pixelsis complete. Accordingly, the process illustrated in FIG. 7 c can bereferred to as a modulo three process.

For each modulo transmission process, the transmission of image datafrom selected rows in each active focal plane can continue until all ofthe pixels in all of the focal planes are transmitted. In embodimentswhere different modulo transmission processes are supported, the imagerarray can provide additional data indicating the modulo process that isbeing utilized. In addition, the imager array can provide additionaldata describing the image data to enable reconstruction of images fromthe image data including (but not limited to) additional data indicatingthe active focal planes and/or information with respect to each line ofimage data specifying the selected rows (columns) in each active focalplane from which the image data is being read out. Although specifictransmission processes are discussed above, image data can betransmitted in any order in accordance with embodiments of theinvention, and is not limited to interleaving image data from multiplefocal planes in a modulo transmission process.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as an example of one embodiment thereof. It istherefore to be understood that the present invention may be practicedotherwise than specifically described, without departing from the scopeand spirit of the present invention. Thus, embodiments of the presentinvention should be considered in all respects as illustrative and notrestrictive.

What is claimed is:
 1. A method of transmitting image data, comprising:capturing image data using a plurality of active focal planes in acamera module, where an image is formed on each active focal plane by aseparate lens stack; generating lines of image data by interleaving theimage data captured by the plurality of active focal planes; andtransmitting the lines of image data and the additional data.
 2. Themethod of claim 1, further comprising generating additional dataproviding information for deinterleaving a plurality of images from thecaptured image data and transmitting the additional data with the linesof image data.
 3. The method of claim 2, wherein the additional datadescribes the imager array.
 4. The method of claim 3, wherein theadditional data includes data selected from the group consisting ofadditional data describing: the number of focal planes in the imagerarray; the dimensions of the array of focal planes in the imager array;the number of pixels in each dimension of the active focal planes; theintegration time of the pixels in each of the active focal planes; andthe gain of the pixels in each of the active focal planes.
 5. The methodof claim 4, wherein the additional data describing the gain of pixelsincludes additional data describing the analog gain of the pixels ineach of the active focal planes and additional data describing thedigital gain of the pixels in each of the active focal planes.
 6. Themethod of claim 2, wherein generating the lines of image data furthercomprises combining captured image data from the plurality of activefocal planes using a predetermined process.
 7. The method of claim 6,wherein: the predetermined process is selected from a plurality ofpredetermined processes for combining captured image data; and theadditional data includes additional data indicating the predeterminedprocess used to combine the captured image data.
 8. The method of claim7, wherein: each active focal plane comprises an array of pixelsincluding a plurality of rows of pixels that also form a plurality ofcolumns of pixels; and the predetermined process for combining capturedimage data comprises interleaving image data from a row selected fromeach of the active focal planes.
 9. The method of claim 7, wherein thepredetermined process for combining captured image data comprises usinga modulo process, where the modulo process involves combining the imagedata captured by each of the pixels in the selected rows of the activefocal planes by interleaving predetermined numbers of pixels from eachof the selected rows.
 10. The method of claim 2, wherein the additionaldata describes at least one of the lines of image data.
 11. The methodof claim 10, wherein the additional data includes data selected from thegroup consisting of additional data describing: the time at which eachactive focal plane started capturing the image data; and the row numbersof the rows in each of the active focal planes used to capture the imagedata used to generate the line of image data.
 12. The method of claim 2,further comprising: packetizing the lines of image data and theadditional data describing the captured image data to produce at leastone packet of data; and wherein transmitting the lines of image data andthe additional data further comprises transmitting the at least onepacket of data.
 13. The method of claim 12, wherein: the packet of datacomprises a packet header containing at least a portion of theadditional data and a plurality of lines of image data; and theadditional data contained in the packet header describes the imagerarray.
 14. The method of claim 12, wherein the packet of data furthercomprises a packet footer indicating the end of the transmission of theplurality of lines of image data.
 15. The method of claim 12, wherein:the packet of data comprises a plurality of lines of image data and aline header associated with each of the plurality of lines of imagedata; and each line header includes additional data describing the lineof image data with which the line header is associated.
 16. The methodof claim 15, wherein the line head further comprises additional datadescribing the imager array.
 17. The method of claim 16, furthercomprising: pausing transmission during frame blanking intervals;transmitting the plurality of lines of image data and the additionaldata describing the captured image data between frame intervals; andpausing during the transmission of the plurality of lines of image dataand the additional data describing the captured image data during lineblanking intervals.
 18. The method of claim 2, further comprisingcontinuously transmitting the lines of image data and the additionaldata until all of the data is transmitted.
 19. The method of claim 1,wherein each lens stack in the camera module has a different field ofview.
 20. A method of receiving image data captured by a camera module,where the camera module captures image data using pixels in a pluralityof focal planes, the method comprising: receiving image data from acamera module including a plurality of focal planes using interfacecircuitry, where the image data comprises lines of image data generatedby interleaving pixels from a plurality of images captured using theplurality of focal planes; and identifying the pixels in the image datathat are part of at least one of the plurality of images using theinterface circuitry.